I. IntroductionII. Scope of the evaluationIII. Target of evaluationIV. Basic functionalitiesA. PerceptibilityB. Level of reliabilityC. CapacityD. SpeedE. Statistical undetectabilityF. AsymmetryV. EvaluationA. PerceptibilityB. Reliability1) Robustness2) False positivesC. CapacityD. SpeedE. Statistical undetectabilityVI. Methodology – Need for third partyVII. Conclusions and future workVIII. ReferencesAbstract—Digital watermarking has beenpresented as a solution to copy protectionof multimedia objects and dozens ofschemes and algorithms have beenproposed. Two main problems seriouslydarken the future of this technologythough. Firstly, the large number of attacks andweaknesses which appear as fast as newalgorithms are proposed, emphasizes thelimits of this technology and in particularthe fact that it may not match usersexpectations.Secondly, the requirements, tools andmethodologies to assess the currenttechnologies are almost non-existent. Thelack of benchmarking of currentalgorithms is blatant. This confusesrights holders as well as software andhardware manufacturers and preventsthem from using the solution appropriateto their needs. Indeed basing long-livedprotection schemes on badly testedwatermarking technology does not makesense.In this paper we will discuss how onecould solve the second problem by havinga public benchmarking service. We willexamine the challenges behind such aservice.Index Terms—watermarking, robustness,evaluation, benchmarkI. INTRODUCTIONDigital watermarking remains alargely untested field and only veryfew large industrial consortiums havepublished requirements againstwhich watermarking algorithmsshould be tested [1, 2]. For instancethe International Federation for thePhonographic Industry led one of thefirst large scale comparative testingof watermarking algorithm for audio.In general, a number of broad claimshave been made about the‘robustness’ of various digitalwatermarking or fingerprintingmethods but very few researchers orcompanies have published extensivetests on their systems.The growing number of attacksagainst watermarking systems (e.g.,[3, 4, 5]) has shown that far moreresearch is required to improve thequality of existing watermarkingmethods so that, for instance, thecoming JPEG 2000 (and newmultimedia standards) can be morewidely used within electroniccommerce applications.We already pointed out in [6] thatmost papers have used their ownlimited series of tests, their ownpictures and their own methodologyand that consequently comparisonwas impossible without re-implementing the method and tryingto test them separately. But then, theimplementation might be verydifferent and probably weaker thanthe one of the original authors. Thisled to suggest that methodologies forevaluating existing watermarkingalgorithms were urgently requiredand we proposed a simple benchmarkfor still image marking algorithms.With a common benchmark authorsand watermarking software providerswould just need to provide a more orless detailed table of results, whichwould give a good and reliablesummary of the performances of theproposed scheme. So end users cancheck whether their basicrequirements are satisfied,researchers can compare differentalgorithms and see how a method canWatermarking schemes evaluationFabien A. P. Petitcolas, Microsoft [email protected] improved or whether a newlyadded feature actually improves thereliability of the whole method andthe industry can properly evaluaterisks associated to the use of aparticular solution by knowing whichlevel of reliability can be achieved byeach contender. Watermarkingsystem designers can also use suchevaluation to identify possible weakpoints during the early developmentphase of the system.Evaluation per se is not a newproblem and significant work hasbeen done to evaluate, for instance,image compression algorithms orsecurity of information systems [7]and we believe that some of it may bere-used for watermarking.In section II will explain what is thescope of the evaluation we envisage.Section III will review the type ofwatermarking schemes that anautomated evaluation service1 coulddeal with. In section IV we will reviewwhat are the basic functionalities thatneed to be evaluated. Section V willexamine how each functionality canbe tested. Finally, section VI willargue the need for a third partyevaluation service and briefly sketchits architecture.II. SCOPE OF THE EVALUATIONWatermarking algorithms are oftenused in larger system designed toachieve certain goals (e.g.,prevention of illegal copying). Forinstance Herrigel et al. [8] presenteda system for trading images; thissystem uses watermarkingtechnologies but relies heavily oncryptographic protocols.Such systems may be flawed for otherreasons than watermarking itself; forinstance the protocol, which uses thewatermark2, may be wrong or therandom number generator used bythe watermark embedder may not be1 Such service is the logical continuationof the existing StirMark benchmark.good. In this paper we are onlyconcerned with the evaluation ofwatermarking (so the signalprocessing aspects) within the largersystem not the effectiveness of thefull system to achieve its goals.III. TARGET OF EVALUATIONThe first step in the evaluationprocess is to clearly identify thetarget of evaluation, that is thewatermarking scheme (set ofalgorithms required for embeddingand extraction) subject to evaluationand its purpose. The purpose of ascheme is defined by one or moreobjectives and an operationalenvironment. For instance, we maywish to evaluate a watermarkingscheme that allows automaticmonitoring of audio tracks broadcastover radio.Typical objectives found across