How to compare two log files and find where a run went wrong

The example compares the end of two worker logs, with the timestamps already removed. The first three lines match, so they show as context. Then the runs split: the first one retried job 381 after a timeout, while the second failed it outright and failed job 977 too. The tool reports 1 change: 2 lines removed, 3 added. The retry and error lines are mostly different, so they show without word marks. In the last line, the marks land on the job count and the run time.

Real logs need some cleanup before a diff is useful. Almost every log line starts with a timestamp, so two runs never match line for line. Strip it first. If each line starts with an ISO 8601 time and a space, this removes it:

sed -E 's/^[0-9T:.+-]+Z? //' run-1.log > run-1.clean.log

Other values change from run to run too: process ids, request ids, durations and temporary file names. Replace each of them with a fixed word, like sed -E 's/request_id=[a-z0-9-]+/request_id=X/', so they stop showing up as changes.

The tool takes up to 5 MB per side, which is tens of thousands of typical log lines. Big texts are compared in a background worker, so the page stays responsive while it runs. For anything past that, cut the logs down to the time window that matters with grep or sed first.