[cpp] CPD gives wrong duplication blocks for CPP code

[NileshVirkar]

Description:
I have ran cpdgui.bat(version 5.7.0) for two cpp files, but got wrong duplication blocks.

Please find the attached cpp source code. (pls remove .txt extension)

HttpSM.cc.txt
HttpSM.h.txt

Result:
Results.txt

Running PMD through: [ext|gui]

[jsotuyod]

@NileshVirkar that's very interesting. Are you using any extra config?

[NileshVirkar]

Hi @jsotuyod, No I didn't use any extra configuration, Pls check the screenshot below.

Thanks!

[adangel]

I could reproduce it and it seems, this bug is in PMD already a long time .... to be exact, since 5.0.1.

The result for PMD 5.0.0 is this, which looks correct:

Found a 7 line (89 tokens) duplication in the following files: 
Starting at line 3814 of /home/andreas/PMD/source/pmd-it/issue-431/HttpSM.cc
Starting at line 4414 of /home/andreas/PMD/source/pmd-it/issue-431/HttpSM.cc

      Debug("http", "[%" PRId64 "] hostdb update marking IP: %u.%u.%u.%u (port %d) as up",
            sm_id,
            ((unsigned char *) &t_state.current.server->ip)[0],
            ((unsigned char *) &t_state.current.server->ip)[1],
            ((unsigned char *) &t_state.current.server->ip)[2],
            ((unsigned char *) &t_state.current.server->ip)[3], t_state.current.server->port);
    }
=====================================================================
Found a 5 line (77 tokens) duplication in the following files: 
Starting at line 3815 of /home/andreas/PMD/source/pmd-it/issue-431/HttpSM.cc
Starting at line 4396 of /home/andreas/PMD/source/pmd-it/issue-431/HttpSM.cc
Starting at line 4415 of /home/andreas/PMD/source/pmd-it/issue-431/HttpSM.cc

            sm_id,
            ((unsigned char *) &t_state.current.server->ip)[0],
            ((unsigned char *) &t_state.current.server->ip)[1],
            ((unsigned char *) &t_state.current.server->ip)[2],
            ((unsigned char *) &t_state.current.server->ip)[3], t_state.current.server->port);

I'm not sure about line 4396 - since it is not sm_id... not sure, how we handle strings here.
The two duplications are the same, by the way: the first is just longer.

Looking at the commit 5.0.0..5.0.1, it seems, that we introduced this bug while fixing sf bug 1023: adding support for continuation characters.

Removing the ContinuationReader and analyzing the two files gives the correct result again. And there is no parsing/lexer error, so it seems, that these files do not really contain multiline-strings, but our ContinuationReader seems to find some. It filters out the backslashes and newline characters - thus maybe leading to wrong linenumbers/columns/position.

[adangel]

Just for reference, other related (old) bugs/threads:

• #1023 c/c++ \ as a continuation character not supported
• Parsing LLVM
• #120 C++ Multi line string literal fix
• #742 CPD C++ lex bug
• #616 CPD lex bug