TIL - Jenkins Iterators
This past week I got bit by a couple of issues with using collections in a Jenkins pipeline that are, in my opinion, violations of the Principle of Least Surprise. One of them is related to the need for all objects in a Jenkins pipeline to be serializable, and the other is related to how Java/Groovy treats references to objects. Both of them cost me a significant amount of time, so hopefully this blog post will save someone else the same pain and suffering.
In my situation, I had a Groovy map with some configuration values that I needed for my pipeline. I was using these values to construct a set of closures that I was going to use with the
parallel keyword. My pipeline code looks something like this:
Issue 1: Iterators Are Not Serializable
The first error I encountered was that iterators aren’t serializable. Jenkins expects everything to be serializable so that it can save state so that it can restart a pipeline if execution is interrupted, and so it can ship the various stages off to secondary processing nodes. The workaround here is to create an external function to convert the map iterator into a list, and mark it with the
@NonCPS attribute. This attribute tells Jenkins that it must execute this function all together. For this purpose, I created a small utility function:
Issue 2: Iterators Use Copy-By-Reference
The second issue did not cause an error, but did result in unexpected behavior. By default, objects in Java are copy-by-reference. In the case of the iterator, when you use the contents of the
CONFIG collection, you are getting a reference to the iterator object instead of a copy of the map values. In the case of deferred execution, it means the iterator is pointed at the last item in the collection. Instead of executing the three individual tasks as expected, it instead executes the last task three times. To avoid this unexpected behavior, you need to force it to copy the values instead of the reference. In my case I wrapped the configuration values in a
String object. A
String in Java is immutable, so it will perform a deep copy when creating the object.
With the changes described above, the final version of my code look like this:
This version of the code executes the three tasks in parallel as expected.
I was pretty frustrated that it took me the better part of a day to figure out the iterator issue. I will admit that I do not have much experience with Java, and after this I cannot say I have any real interest in learning more about it. Hopefully this will save somebody else the time and frustration if they encounter these same issues.