TIL - Jenkins Iterators

Developer Journal
,

Jenkins Iterators

Introduction

This past week I got bit by a couple of issues with using collections in a Jenkins pipeline that are, in my opinion, violations of the Principle of Least Surprise. One of them is related to the need for all objects in a Jenkins pipeline to be serializable, and the other is related to how Java/Groovy treats references to objects. Both of them cost me a significant amount of time, so hopefully this blog post will save someone else the same pain and suffering.

The Setup

In my situation, I had a Groovy map with some configuration values that I needed for my pipeline. I was using these values to construct a set of closures that I was going to use with the parallel keyword. My pipeline code looks something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#!groovy

CONFIG = [
    "123", "task1",
    "456", "task2",
    "789", "task3"
]

pipeline {
    agent any
    stages {
        stage("Run parallel tasks")
            steps {
                script {
                    def tasks = [:]
                    for(t in CONFIG) {
                        tasks["Task - ID: ${t[0]}, Name: ${t[1]}] = {
                            // Task code goes here
                        }
                    }

                    parallel tasks
                }
            }
        }
    }
}

Issue 1: Iterators Are Not Serializable

The first error I encountered was that iterators aren’t serializable. Jenkins expects everything to be serializable so that it can save state so that it can restart a pipeline if execution is interrupted, and so it can ship the various stages off to secondary processing nodes. The workaround here is to create an external function to convert the map iterator into a list, and mark it with the @NonCPS attribute. This attribute tells Jenkins that it must execute this function all together. For this purpose, I created a small utility function:

1
2
3
4
5
6
@NonCPS
List<List<?>> mapToList(Map map) {
  return map.collect { item ->
    [item.key, item.value]
  }
}

Issue 2: Iterators Use Copy-By-Reference

The second issue did not cause an error, but did result in unexpected behavior. By default, objects in Java are copy-by-reference. In the case of the iterator, when you use the contents of the CONFIG collection, you are getting a reference to the iterator object instead of a copy of the map values. In the case of deferred execution, it means the iterator is pointed at the last item in the collection. Instead of executing the three individual tasks as expected, it instead executes the last task three times. To avoid this unexpected behavior, you need to force it to copy the values instead of the reference. In my case I wrapped the configuration values in a String object. A String in Java is immutable, so it will perform a deep copy when creating the object.

Final Version

With the changes described above, the final version of my code look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#!groovy

CONFIG = [
    "123", "task1",
    "456", "task2",
    "789", "task3"
]

pipeline {
    agent any
    stages {
        stage("Run parallel tasks")
            steps {
                script {
                    def tasks = [:]
                    for(t in CONFIG) {
                        String id = t[0]
                        String name = t[1]
                        tasks["Task - ID: ${id}, Name: ${name}] = {
                            // Task code goes here
                        }
                    }

                    parallel tasks
                }
            }
        }
    }
}

@NonCPS
List<List<?>> mapToList(Map map) {
  return map.collect { item ->
    [item.key, item.value]
  }
}

This version of the code executes the three tasks in parallel as expected.

Conclusion

I was pretty frustrated that it took me the better part of a day to figure out the iterator issue. I will admit that I do not have much experience with Java, and after this I cannot say I have any real interest in learning more about it. Hopefully this will save somebody else the time and frustration if they encounter these same issues.