Global Fight Against Log4j Vulnerability Relies on Apache Volunteers

Crucial to the effort are Mr. Gregory and four other Apache volunteers, all of whom hold day jobs. In recent days, they have scrambled to release updates to Log4j and work with businesses to mitigate the looming threat.

Apache, a nonprofit that distributes the open-source tool at no cost, has said it has been downloaded millions of times. Log4j is used on computer servers to keep records of users’ activities and applications’ behaviors so they can be reviewed later by security or software development teams. The vulnerability could allow hackers to remotely execute code that takes over devices or infects them with malware.

Mr. Gregory, who works from the dining-room table in his Ocala, Fla., home, fueled by black coffee and accompanied by his hound-pit-bull mix, Bella, said he is overwhelmed with hundreds of requests for help from businesses. While Apache is trying to assist companies in updating their systems, he said, the nonprofit’s resources are limited.

“This puts to the forefront the whole issue with open-source [software] and commercial users,” said Mr. Gregory, who is on the Apache Logging Services Project Management Committee of 16 elected members who vote on changes to the software. “The expectations are somewhat out of whack.”

Mr. Gregory, whose day job is principal software engineer at Massachusetts-based Rocket Software Inc., was helping to finalize a security update for Log4j last week when an email blew up his plans.

A tipster had alerted Apache volunteers to the security flaw in late November, prompting them to work on a patch. Last Thursday, a day before Apache was set to release the patch, the same tipster said in an email that users on Chinese chat forums were already discussing the vulnerability.

“We very quickly realized that this was dramatic and dangerous,” Mr. Gregory said. Or to put it another way, he added, “Holy crap, this is bad.”

Many developers rely on the free Log4j framework to help record data such as users’ behavior and applications’ activity in software built with the Java programming language. Cybersecurity experts say the inclusion of the open-source logging tool within so much interconnected software—often embedded without developers’ knowledge—yields a threat that spans economic sectors and national borders.

“This is an everywhere problem,” said Theresa Payton, former White House chief information officer and chief executive of cyber consulting firm Fortalice Solutions LLC.

In Germany, the security team at chemicals company Evonik Industries AG hurried to pinpoint Log4j in its network and disabled an online learning application for employees as a precaution. Milwaukee, Wis.-based industrial-parts supplier Rockwell Automation Inc. rushed to communicate with vendors about their own exposure to the flaw. U.S. tech companies such as International Business Machines Corp. and VMware Inc. said they are deploying patches.

A partnership recently launched by the U.S. Cybersecurity and Infrastructure Security Agency, cloud-service providers such as Amazon.com Inc. and telecom companies including Verizon Communications Inc. has held daily calls to share information about potential threats, according to a person familiar with the matter. CISA officials said on a separate call with critical-infrastructure operators on Monday that hundreds of millions of devices could be at risk.

As businesses update their systems and probe vendors for vulnerabilities, cybersecurity company Mandiant Inc. said it has observed Chinese government hackers trying to exploit the flaw.

Matthew Prince, chief executive of Cloudflare Inc., which has wide visibility of cloud-computing infrastructure, warned of increasingly dangerous hacking attempts.

“Ransomware payloads started in force in [the] last 24 hours,” Mr. Prince wrote on Twitter on Tuesday. Cybersecurity experts haven’t tied a specific successful ransomware attack to the Log4j vulnerability.

After Apache released its planned patch on Friday, Mr. Gregory said he worked through the weekend on a new update along with other volunteer software developers in Japan, New Zealand, Virginia and Arizona. Unveiled Monday, the new version disabled a problematic software module by default and removed a message-lookup feature that could be used to exploit the flaw.

The Apache volunteers are designing another update to Log4j for users who rely on an older version of the Java programming language, meaning more work for Mr. Gregory while he is on vacation from his day job.

“That translates to me getting five hours of sleep last night,” he said of his time off. “Some of the other guys got two or three.”

Write to David Uberti at [email protected]