This blog was published by Frédéric Combes and translated into English for the Guardsquare Community. Make sure to check out his original post, his contributions at i-Run, and the rest of his work here.
A while ago I saw this video from Nicolas Peters at Devoxx talking about picocli and java command line applications. At i-run we have big DevOps tooling needs for our new projects and suddenly we wanted to test picocli to develop some tools.
It works fine, but over time and dependencies the binary has gained weight and even keeping the dependencies to the bare minimum the jar is 1.2MB today (just the start). We therefore sought a way to limit the size of the binary so that it remains easy to deploy on the machines which need it and that the memory borrowing is as small as possible. We came naturally to ProGuard .
ProGuard
ProGuard is well known within the framework of Android for its ability to reduce and especially obfuscate code. But we think less about it in standard java projects. Yet it perfectly meets the need we have at I-Run.
The problem is that ProGuard is not an intuitive tool and that it is quite easy to break your build with it. Especially since most of the explanations or examples, present on the internet, relate to Android projects.
I suggest you see the configuration that we have set up at I-Run for our tool.
The principles
ProGuard will optimize a jar by going through several processing phases that are important to understand.
shrink : Which will prune anything that is not used in your code.
optimize : Which will go a little further by removing dead code and method parameters for example.
obfuscate : Who will rename the names of classes, methods, members,… and reduce their names to a minimum (a letter or two).
And there we understand the power of the tool but also the problems that we will encounter!
The result
In the end, with optimization and obfuscation, the binary went from 1.2 MB to 668 KB or more than 50% of the initial size.
Project description
The project on which we are going to apply ProGuard is a command line tool. We limit the dependencies to the maximum to control the size of the final jar. The application is packaged in the form of an Uber jar via the maven shade plugin.
We therefore use
- picocli to parse the command line
- flogger for logs
- feather for dependency injection
- type-safe / config for configuration management
- nano-json for reading and writing json
First attempt
Already the minimum configuration for that to work, it is necessary to indicate where are the libraries with -libraryjars and the input / output -injars -outjars .
-injars coffees/target/coffees-1.3.0-SNAPSHOT-shaded.jar(!META-INF/versions/**)
-outjars coffees/target/coffees-1.3.0-SNAPSHOT-slim.jar
-libraryjars <java.home>/jmods/(!**.jar;!module-info.class) # Pour Java 11
-libraryjars <java.home>/lib/rt.jar # Pour Java 8
The first launch displays pages and pages of warnings and notes to end in error.
ProGuard, version 6.2.0
Reading program jar [/home/marthym/workspace/i-run/coffee-shell/coffees/target/coffees-1.4.0-SNAPSHOT-shaded.jar] (filtered)
Reading library directory [/opt/jdk-jdk-11.0.5+10/jmods] (filtered)
Warning: com.google.common.flogger.AbstractLogger: can't find referenced class com.google.errorprone.annotations.CheckReturnValue
Warning: com.google.common.flogger.FluentLogger: can't find referenced class com.google.errorprone.annotations.CheckReturnValue
Warning: com.google.common.flogger.LazyArg: can't find referenced class javax.annotation.Nullable
Warning: com.google.common.flogger.LogContext: can't find referenced class javax.annotation.Nullable
Warning: com.google.common.flogger.LogContext: can't find referenced class javax.annotation.Nullable
Warning: com.google.common.flogger.LogContext: can't find referenced class javax.annotation.Nullable
Warning: com.google.common.flogger.LogContext: can't find referenced class javax.annotation.Nullable
....
Maybe this is library field 'java.lang.ProcessBuilder$Redirect$Type { java.lang.ProcessBuilder$Redirect$Type INHERIT; }'
Note: there were 9 classes trying to access annotations using reflection.
You should consider keeping the annotation attributes
(using '-keepattributes *Annotation*').
(http://proguard.sourceforge.net/manual/troubleshooting.html#attributes)
Note: there were 11 classes trying to access generic signatures using reflection.
You should consider keeping the signature attributes
(using '-keepattributes Signature').
(http://proguard.sourceforge.net/manual/troubleshooting.html#attributes)
Note: there were 4 unresolved dynamic references to classes or interfaces.
You should check if you need to specify additional program jars.
(http://proguard.sourceforge.net/manual/troubleshooting.html#dynamicalclass)
Note: there were 38 accesses to class members by means of reflection.
You should consider explicitly keeping the mentioned class members
(using '-keep' or '-keepclassmembers').
(http://proguard.sourceforge.net/manual/troubleshooting.html#dynamicalclassmember)
Warning: there were 873 unresolved references to classes or interfaces.
You may need to add missing library jars or update their versions.
If your code works fine without the missing classes, you can suppress
the warnings with '-dontwarn' options.
(http://proguard.sourceforge.net/manual/troubleshooting.html#unresolvedclass)
Error: Please correct the above warnings first.
We can already add to the configuration the fact of not stopping in the event of a warning.
-dontwarn
At this point we still have no results, instead a message telling us that we have to keep something for it to work.
Error: You have to specify '-keep' options if you want to write out kept elements with '-printseeds'.
We must give ProGuard the entry point of the program, in our case the main program. The configuration of ProGuard resembles the syntax of java.
-keep class fr.irun.coffee.shell.CoffeesMainCommand {
public static void main(java.lang.String[]);
}
attention it is imperative to use the full name of the classes.
This time we finally have a result. A result which is very likely not to work, but a result. We now need to improve the configuration to make everything work.
Annotations
As proguard tries to optimize the code, by default it will not keep the annotations. So if your code uses it at runtime, it won’t work anymore. We will be able to tell him to keep the annotations with this instruction.
-keepattributes *Annotation*, Signature, Exception
In the end ProGuard will keep a little more than the annotations. It will also keep the generic types of method signatures and the exceptions that a method can throw.
Another problem with annotations is that picocli instantiates itself the classes of so Command , as for the injection, ProGuard has no way of knowing that these classes are being used unless it is asked to keep them. For that :
-keep class picocli.CommandLine { *; }
-keep class picocli.CommandLine$* { *; }
-keepclassmembers class * extends java.util.concurrent.Callable {
public java.lang.Integer call();
}
With the first two lines, it is specified to keep all the classes that are annotated with CommandLine or one of the annotation subclasses. With the rest we ask it not to remove the methods call() on the classes that inherit from Callable .
Dependency injection
Dependency injection breaks the thinking that ProGuard does by going up the classes used. As we do not explicitly declare the instantiation of the class, it cannot know the implementation.
To overcome this problem, we can add this configuration
# Keep for class injection
-keepclassmembers class * {
@org.codejargon.feather.Provides *;
@javax.inject.Inject <init>(...);
@picocli.CommandLine$Option *;
}
Everything that is annotated Inject or Provides , for dependency injection, is kept as is, no name change or deletion. Ditto for CommandLine$Option which is used by picocli to inject values into class members.
Plugin management
Dependency injection is used to manage plugins. Each subcommand is a plugin that implements a module interface CoffeeModule . The search for implementations is done via one ServiceLoader which is configured with files META-INF/services/... in which the implementations are listed.
If we let ProGuard do it, it will obfuscate the classes, change the names and suddenly, the configuration of our services will not work. To avoid this we add the following instructions:
# Keep class used for plugin management
-keepnames class fr.irun.coffee.swizzle.plugin.CoffeeModule
-keepnames class * implements fr.irun.coffee.swizzle.plugin.CoffeeModule
-keep class * implements fr.irun.coffee.swizzle.plugin.CoffeeModule {
java.util.List getModuleCommands();
}
In order :
- We do not change the names of CoffeeModule
- We do not change the names of the classes that implement CoffeeModule
- We do not change the name of the interface methods of CoffeeModule
The problem of enums
During the optimization phase, ProGuard optimizes the enums to integer constants, “when possible”. Obviously, when it’s not possible, he does it anyway. So sometimes we lose some functionality. To prevent this optimization from breaking your enums:
# Fix enum problems
-keepclassmembers class * extends java.lang.Enum {
<fields>;
public static **[] values();
public static ** valueOf(java.lang.String);
}
Removal of remaining warnings
At this point, most of the work is done. To go to the end and finish the job properly, it remains to check and delete, one by one, the remaining warnings. It’s faster than it looks!
can’t find referenced class
Warning: com.google.common.flogger.LazyArg: can't find referenced class javax.annotation.Nullable
Warning: com.google.common.flogger.LogContext: can't find referenced class javax.annotation.Nullable
Warning: com.google.common.flogger.LogContext: can't find referenced class javax.annotation.Nullable
Warning: com.google.common.flogger.LogContext: can't find referenced class javax.annotation.Nullable
These kinds of warnings are not important and can be safely ignored. These are most often annotations useful only for compilation and whose code is in dependencies provided .
-dontwarn org.fusesource.**
-dontwarn lombok.**
-dontwarn javax.annotation.**
-dontwarn com.google.errorprone.**
accesses a declared method dynamically
The bulk of the messages are displayed. ProGuard tells us that he sees that a piece of code is accessing a method by reflection and that he does not know which class the method belongs to. It then lists all the candidate classes it finds in the classpath. And there are often many.
Note: picocli.CommandLine$Interpreter accesses a declared method 'parse(java.lang.CharSequence)' dynamically
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$5 { java.util.Date parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$5 { java.lang.Object parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$6 { java.io.File parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$6 { java.lang.Object parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$7 { java.net.URL parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$7 { java.lang.Object parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$8 { java.net.URI parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$8 { java.lang.Object parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$9 { java.lang.Class parse(java.lang.CharSequence); }'
Maybe this is library method 'com.sun.xml.internal.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl$9 { java.lang.Object parse(java.lang.CharSequence);
...
Maybe this is library method 'java.time.Duration { java.time.Duration parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.Instant { java.time.Instant parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.LocalDate { java.time.LocalDate parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.LocalDateTime { java.time.LocalDateTime parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.LocalTime { java.time.LocalTime parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.MonthDay { java.time.MonthDay parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.OffsetDateTime { java.time.OffsetDateTime parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.OffsetTime { java.time.OffsetTime parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.Period { java.time.Period parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.Year { java.time.Year parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.YearMonth { java.time.YearMonth parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.ZonedDateTime { java.time.ZonedDateTime parse(java.lang.CharSequence); }'
Maybe this is library method 'java.time.format.DateTimeFormatter { java.time.temporal.TemporalAccessor parse(java.lang.CharSequence); }'
Here, to correct, you have to go to the code and see that it is the class a which belongs to the method called by reflection. Then the easiest way is to copy the end of the message after the Maybe which corresponds to the correct class and add -keep class .
-keep class java.sql.DriverManager { java.sql.Connection getConnection(java.lang.String); }
In the case of java.time , it is picocli which uses them for several types of different class. To do them all at once we can use this configuration
-keep class java.time.** {
public java.time.** parse(java.lang.CharSequence);
public java.time.** of(java.lang.String);
}
It does not rename parse or of in the package classes java.time because picocli is used in reflection.
For the remaining notes
For the few remaining notes that do not impact the code:
-dontnote com.google.common.flogger.**
-dontnote com.typesafe.config.**
Final configuration
At the end the configuration file looks like this
-injars coffees/target/coffees-1.3.0-SNAPSHOT-shaded.jar(!META-INF/versions/**)
-outjars coffees/target/coffees-1.3.0-SNAPSHOT-slim.jar
# For java 10+
#-libraryjars <java.home>/jmods/java.base.jmod(!**.jar;!module-info.class)
-libraryjars <java.home>/lib/rt.jar
#-dontobfuscate
-optimizationpasses 5
-printmapping coffeer.map
-keepattributes *Annotation*, Signature, Exception
# Need to fix bug in ProGuard prio than 6.2
#-optimizations !code/allocation/variable # Fix bug with VTL
# Keep for class injection
-keepclassmembers class * {
@org.codejargon.feather.Provides *;
@javax.inject.Inject <init>(...);
@picocli.CommandLine$Option *;
}
# Keep classes for picocli command line parsing
-keep class picocli.CommandLine { *; }
-keep class picocli.CommandLine$* { *; }
-keep class fr.irun.coffee.shell.CoffeesMainCommand {
public static void main(java.lang.String[]);
}
-keepclassmembers class * extends java.util.concurrent.Callable {
public java.lang.Integer call();
}
# Keep class used for plugin management
-keepnames class fr.irun.coffee.swizzle.plugin.CoffeeModule
-keepnames class * implements fr.irun.coffee.swizzle.plugin.CoffeeModule
-keep class * implements fr.irun.coffee.swizzle.plugin.CoffeeModule {
java.util.List getModuleCommands();
}
# Fix enum problems
-keepclassmembers class * extends java.lang.Enum {
<fields>;
public static **[] values();
public static ** valueOf(java.lang.String);
}
# jline dynamic
-keep class java.lang.ProcessBuilder$Redirect { java.lang.ProcessBuilder$Redirect INHERIT; }
-keep class java.time.** {
public java.time.** parse(java.lang.CharSequence);
public java.time.** of(java.lang.String);
}
-keep class java.sql.DriverManager { java.sql.Connection getConnection(java.lang.String); }
-keep class java.sql.DriverManager { java.sql.Driver getDriver(java.lang.String); }
-keep class java.nio.file.Paths { java.nio.file.Path get(java.lang.String,java.lang.String[]); }
-keep class java.io.Console { char[] readPassword(java.lang.String,java.lang.Object[]); }
-keep class java.lang.reflect.Parameter { java.lang.String getName(); }
-dontnote com.google.common.flogger.**
-dontnote com.typesafe.config.**
-dontnote jline.OSvTerminal
-dontnote picocli.CommandLine$Help$Ansi
-dontnote java.time.**
-dontnote java.sql.DriverManager
-dontnote java.nio.file.Paths
-dontnote java.io.Console
-dontnote java.lang.reflect.Parameter
-dontnote java.lang.ProcessBuilder$Redirect
-dontnote fr.irun.coffee.modules.**
-dontnote fr.irun.coffee.shell.helpers.GitlabTokenValidator
-dontnote javax.inject.Provider
-dontnote fr.irun.coffee.swizzle.options.GitlabConfigOptions
-dontnote fr.irun.coffee.swizzle.config.Configuration
-dontwarn org.fusesource.**
-dontwarn lombok.**
-dontwarn javax.annotation.**
-dontwarn com.google.errorprone.**
Have questions or comments? Let us know below or on Frédéric’s original blog version