Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-49584][BUILD] Upgrade log4j2 to 2.24.0 #48057

Closed
wants to merge 5 commits into from
Closed

[SPARK-49584][BUILD] Upgrade log4j2 to 2.24.0 #48057

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
aaf8c87
init
LuciferYang Sep 10, 2024
b6b26cf
workaround
LuciferYang Sep 10, 2024
312a3a6
fix format
LuciferYang Sep 10, 2024
a7280c0
revert
LuciferYang Sep 10, 2024
2a65692
Merge branch 'apache:master' into log4j-2.24.0
LuciferYang Sep 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions common/utils/src/test/resources/log4j2.component.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
log4j2.garbagefree.threadContextMap = true
Copy link
Contributor Author

@LuciferYang LuciferYang Sep 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/apache/spark/blob/7355864e2cf8329188e69365415ab0bd718d0c1a/common/utils/src/test/scala/org/apache/spark/util/StructuredLoggingSuite.scala#L468C3-L478

  test("setting to MDC gets logged") {
    val mdcPattern = s""""${LogKeys.DATA.name}":"some-data""""

    org.slf4j.MDC.put(LogKeys.DATA.name, "some-data")
    val logOutputWithMDCSet = captureLogOutput(() => logInfo(msgWithMDC))
    assert(mdcPattern.r.findFirstIn(logOutputWithMDCSet).isDefined)

    org.slf4j.MDC.remove(LogKeys.DATA.name)
    val logOutputWithoutMDCSet = captureLogOutput(() => logInfo(msgWithMDC))
    assert(mdcPattern.r.findFirstIn(logOutputWithoutMDCSet).isEmpty)
  }

There is a significant refactoring in the ThreadContextMap component(apache/logging-log4j2#2330), resulting in the loss of key-value pairs injected via org.slf4j.MDC during the generation of LogEvent objects when using DefaultThreadContextMap. As a workaround, it is necessary to configure and employ GarbageFreeSortedArrayThreadContextMap. I am puzzled by this situation. Since org.slf4j.MDC is a behavior specific to SLF4J, and this does not appear to be an incompatible change for Log4j, let's currently halt this upgrade and wait to see if the new version of SLF4J can address this issue.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @dongjoon-hyun and @panbingkun for awareness

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @gengliangwang , too

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, @LuciferYang , there is a PR already from @panbingkun .

10 changes: 5 additions & 5 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,11 +184,11 @@ lapack/3.0.3//lapack-3.0.3.jar
leveldbjni-all/1.8//leveldbjni-all-1.8.jar
libfb303/0.9.3//libfb303-0.9.3.jar
libthrift/0.16.0//libthrift-0.16.0.jar
log4j-1.2-api/2.22.1//log4j-1.2-api-2.22.1.jar
log4j-api/2.22.1//log4j-api-2.22.1.jar
log4j-core/2.22.1//log4j-core-2.22.1.jar
log4j-layout-template-json/2.22.1//log4j-layout-template-json-2.22.1.jar
log4j-slf4j2-impl/2.22.1//log4j-slf4j2-impl-2.22.1.jar
log4j-1.2-api/2.24.0//log4j-1.2-api-2.24.0.jar
log4j-api/2.24.0//log4j-api-2.24.0.jar
log4j-core/2.24.0//log4j-core-2.24.0.jar
log4j-layout-template-json/2.24.0//log4j-layout-template-json-2.24.0.jar
log4j-slf4j2-impl/2.24.0//log4j-slf4j2-impl-2.24.0.jar
logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
lz4-java/1.8.0//lz4-java-1.8.0.jar
metrics-core/4.2.27//metrics-core-4.2.27.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@
<sbt.project.name>spark</sbt.project.name>
<asm.version>9.7</asm.version>
<slf4j.version>2.0.16</slf4j.version>
<log4j.version>2.22.1</log4j.version>
<log4j.version>2.24.0</log4j.version>
<!-- make sure to update IsolatedClientLoader whenever this version is changed -->
<hadoop.version>3.4.0</hadoop.version>
<!-- SPARK-41247: When updating `protobuf.version`, also need to update `protoVersion` in `SparkBuild.scala` -->
Expand Down