Experiment with running headless Firefox in parallel

[davehunt]

This change runs headless Firefox instances in parallel.

[coveralls]

Coverage remained the same at 100.0% when pulling 77ed37b on davehunt:heracles into 32f1be5 on mozilla:master.

[coveralls]

Coverage decreased (-3.1%) to 96.875% when pulling 6c6e0ba on davehunt:heracles into 32f1be5 on mozilla:master.

[coveralls]

Coverage decreased (-3.1%) to 96.875% when pulling 6c6e0ba on davehunt:heracles into 32f1be5 on mozilla:master.

[coveralls]

Coverage decreased (-3.1%) to 96.875% when pulling 6c6e0ba on davehunt:heracles into 32f1be5 on mozilla:master.

[davehunt]

Interestingly, four processes introduced some failures. One build passed in 3.5 minutes, but the other had connection refused for several tests. I wonder if we're hitting the available port race condition. Let's leave this open for now.

[davehunt]

If we could get the geckodriver log files, we might be able to confirm the port clashing theory.

[coveralls]

Coverage decreased (-3.1%) to 96.875% when pulling 09860ea on davehunt:heracles into 32f1be5 on mozilla:master.

[coveralls]

Coverage decreased (-3.1%) to 96.875% when pulling 09860ea on davehunt:heracles into 32f1be5 on mozilla:master.

[b4handjr]

Seems like it was just esr that failed...I think geckodriver 18 recommends 53 and up now. Maybe We skip the auto runs on <53 ? I think we could do up to n=4 on 53 and up.

[davehunt]

Seems like it was just esr that failed...

No, there was also a nightly job that failed.

I think geckodriver 18 recommends 53 and up now. Maybe We skip the auto runs on <53?

FoxPuppet has committed to supporting Firefox ESR so it would seem we can't upgrade geckodriver for all versions. Perhaps we'll need to download an older version for ESR, but we should probably also mention that in the documentation.

[b4handjr]

FoxPuppet has committed to supporting Firefox ESR so it would seem we can't upgrade geckodriver for all versions. Perhaps we'll need to download an older version for ESR, but we should probably also mention that in the documentation

Seems easy enough with the new build steps

[whimboo]

If we could get the geckodriver log files, we might be able to confirm the port clashing theory.

@davehunt is there a way to just print the output of geckodriver.log at the very end of the job? Or maybe use S3 upload of built artifacts: https://docs.travis-ci.com/user/uploading-artifacts?

[davehunt]

Seems easy enough with the new build steps

Yeah, I'll take a look today.

Or maybe use S3 upload of built artifacts.

This is my preference. I've set it up once before, so I'll look into it.

[whimboo]

I think geckodriver 18 recommends 53 and up now. Maybe We skip the auto runs on <53?

FoxPuppet has committed to supporting Firefox ESR so it would seem we can't upgrade geckodriver for all versions. Perhaps we'll need to download an older version for ESR, but we should probably also mention that in the documentation.

You might not notice this for ESR only but also for release vs. beta, and central. So it might be a good idea to specify a release of geckodriver for a version of Firefox. With the release of 0.18 re recommend Firefox 54 and later.

[davehunt]

I've raised bug 1390311 to set up an S3 bucket for build artifacts.

[coveralls]

Coverage remained the same at 100.0% when pulling f7b129b on davehunt:heracles into 59f048d on mozilla:master.

[coveralls]

Coverage remained the same at 100.0% when pulling f7b129b on davehunt:heracles into 59f048d on mozilla:master.

[coveralls]

Coverage remained the same at 100.0% when pulling f7b129b on davehunt:heracles into 59f048d on mozilla:master.

[coveralls]

Coverage remained the same at 100.0% when pulling f7b129b on davehunt:heracles into 59f048d on mozilla:master.

[coveralls]

Coverage remained the same at 100.0% when pulling 33fdde3 on davehunt:heracles into 1271f26 on mozilla:master.

[davehunt]

See bug 1388465 for the potentially related issue.

[davehunt]

I've opened #146 to enable uploading artifacts to S3, however note that this not work for pull requests from other users due to the potential to expose AWS secrets. Once that has been merged we can rebase this and push to the main project to see the artifacts.

[davehunt]

Rebased and also pushed to https://github.com/mozilla/FoxPuppet/tree/heracles so we can see the artifacts at https://travis-ci.org/mozilla/FoxPuppet/builds/333932386

[b4handjr]

This is looking good!

[davehunt]

@jrbenny35 we're still seeing occaisional failures when Geckodriver attempts to connect to Firefox. It looks like Firefox isn't launching. I have at least ruled out a port clash now that we have additional logging enabled. I'm going to see if I can replicate on a debug build to investigate further. In the meantime, please don't merge this PR.

[b4handjr]

@davehunt yeah I figured so, that is why I didn't merge it.