Pekka Buttler, August 2020
Updated July 2022 (see update notes)
This article summarises JAPB’s principles for those types of articles in which we review or comment on specific types of gear (mostly lenses) as well as detailing the types of articles in which we comment on gear.
Table of contents:
• Article types
• Reviews
• Comparisons
• Data sheets
• Walk-arounds
• General principles
• Uniqueness
• Neutrality
• Subjectivity
• Conclusion
Article types
Reviews
A review should be a critical appraisal. ‘Critical’ does in this context not mean mean expressing adverse or disapproving comments, but instead means “involving an analysis of the faults and merits”. The three central words are analysis, faults and merits.
Faults and merits go together and mean that we will try to find and report all significant characteristics of a piece of gear (or other object of a review) and tell – when we can – whether we consider them to be merits or faults. In some cases, this is simple, such as when a standard zoom has a minimum focusing distance of more than one meter, as there are few situations in which we would not see that as a fault. In other situations it can be very hard, because the design of gear typically involves making tradeoffs. The point is, that we will always mention all significant characteristics – even when we can’t make up our mind about whether they be faults or merits – because for you those characteristics may matter.
The important characteristic of a JAPB review is that we will tell it all: every piece of information we have, which conceivably might be useful to someone. Our reviews are always in depth. That said, you can expect variance in that depth, because sometimes we might do a review based on only tens of hours of hands-on time, whereas sometimes we’ll write up a review of a piece of kit we’ve used for ages. Unavoidably, that will colour our experience, but we’ll be open about that as well. But let us mention one aspect wherein a JAPB review might differ from other reviews.
Reviews are the staple of photographic sites and magazines and many turn to them for buying advice. In doing so are liable to be terribly misled, because a review can always be tweaked to accentuate a lens’ benefits or gloss over its deficiencies. Even the resolution tables shown in many reviews – while seeming coldly objective – are easy to fudge. You need only look at some sites, read some bloggers or look at some vloggers to notice that they rarely say anything really bad about a piece of gear. This is not necessarily because they’ve been paid for a positive review, but because people like to read good words about a piece of kit they own or are considering to purchase.
Obviously (which is why we’ll repeat it) JAPB is not like that. Firstly, we don’t do MTF -based resolution testing – partially because resolution is not the only important thing about lenses, but more importantly because (considering uniqueness) mathematically exact resolution tests are mostly nonsensical. Instead, our reviews embrace also other aspects, such as handling and ergonomics and maybe history or trivia, and when it comes to image quality, we largely let the images speak for themselves (while often helping you analyse what you’re seeing).
“‘But…“, you may ask, “how will I then know whether I should buy lens A or B?” Well, considering sample variation, maybe we should not try to tell you that, but we are aware that – in isolation – every lens may seem like a star or like a dog. That is why – especially in all the most central categories of legacy lenses – we also do comparisons.
Comparisons
Q: “How do you create an elite team?”
A: “You start by telling them that they’re the elite.”
Even we are susceptible to be fooled by the legendary reputation of a lens. That is why we always compare. We do not write a review of a lens without having compared it to a largely similar lens, and we do not say anything good or bad about image quality without being able to compare a shot with a similar shot from another lens.
This is the spirit which we take to the max when doing comparisons. We take a bunch of lenses with similar numerical characteristics (e.g. large aperture 35 mm focal length lenses), and systematically and rigorously compare them. This means that when we comment on a lens’ aperture ring’s lack of intermediate click-stops then another lens has those. More importantly, when we do image quality comparisons, we set up a tripod, and go through the task of taking shots which aim to be as comparable as possible. Then we show you those shots.
The whole point of doing comparisons is to add perspective. The benefit of perspective – for you – is to know what can feasibly be expected of a lens, and to show you that typically one lens (sample) may outperform another lens in one respect, while struggling in another. Because there (typically) is no such thing as an overall best lens (or other piece of kit).
Data sheets
JAPB also publishes data sheets on lenses. Data sheets are short-ish articles that serve two functions:
1) Data sheets introduce a lens, give details on it, and comment on the lens’ history (when that might be of interest to someone).
2) Data sheets discuss adapting the lens to modern digital cameras, hence giving aspiring legacy lens users a detailed guide on the tools they’ll need and what they can expect.
Walk-arounds
Finally, JAPB also offers accounts on using lenses, which we refer to as walk-arounds, because their starting point is often that we mount a lens on our digital camera and walk around, taking shots and making notes. Walk arounds differ from reviews in that they will not critically assess a lens’ faults and merits. Walk-arounds will also not delve into the lens’ specifications or history.
Instead they focus on detailing the “lived experience” of using the lens and offer sample pictures. Walk-arounds are subjective and do not pretend otherwise. Even so, we’re sure they will have value to many reader.
General principles
We at JAPB want to make some principles concerning our reviews and comparisons very clear, because we are are aware that people may use the information on this site in order to make decisions about how the spend their hard-earned money. If reading JAPB has given you information, that you have used in making those decisions, we are both pleased and flattered. But making you spend money is not our purpose.
In any case, let us make some principles regarding what you read on this site as clear as possible. JAPB reviews and comparisons abide by three principles: uniqueness, neutrality and subjectivity. Let us explain what we mean by those. After that, we’ll try to explain why we do reviews and comparisons (and how that unavoidably affects our reviews and comparisons).
Uniqueness
We never pretend to test the design of a lens (or any other type of photographic gear). We only test samples. As any statistician knows, a sample of 1 is an insufficient number to be able to accurately predict for any population beyond one.
Want it or not, sample variation exists even with modern lenses that just rolled out the end of a production line. Don’t take my word for it, look at the research done by Lensrentals.
Then imagine how much worse things were when your legacy lens was produced. Yes, I know that many legacy lens aficionados think that lenses were built better back in the ‘good old days’. Trust me, they were not. Not only do we have more precise control of material inputs today, also our measurement tools and machine tools are more precise. I see some decry the quality of modern lenses. This is utter foolishness. If some modern lenses are of lower quality, it’s not because we could not produce higher quality, it is because some lenses are designed for lower quality (and a lower price point).
Moreover, JAPB does not test factory-fresh lenses. We might throw in a modern lens now and again, just for the sake of comparison, but even then the lens is not really ‘new’. In fact, we own very few lenses that are not at least second-hand. And, sadly, age does matter. Just imagine how much variation might creep in through the decades of ageing, intensive use, and occasional mistreatment.
Therefore, when we put a lens to the test and it underperforms significantly, all we really know is that our sample underperformed (not that the basic design is bad). We might not always remember to say it at every turn, but it’s the truth and we’re aware of it.
Neutrality
We’re not paid ‘influencers’. We do not push any brand or make of lens on you as part of some deal that would make us rich (I sometimes wish we did). If we say that a sample of a Canon lens is very good and beats the living daylights out of a sample of a Nikon lens, then we say say because that is our honest evaluation of what has happened, not because we’re canon fan-boys (or girls).
In fact, unless expressly stated otherwise, all the gear we test is the personal property of the article author – if we borrow a lens from a fellow traveller to round out a comparison, we mention that explicitly. We’re also prepared to review gear sent to us, but only because we can’t afford to buy every lens we’d like to try. We will not promise a review, and we definitely do not promise a favourable review – in fact we might be liable to take a more critical stance towards such items. And we will ALWAYS state if the lens is not personally procured for real money.
Subjectivity
But the fact that we’re neutral does not mean that we would (or, indeed could) be objective. We – just as you – are humans, and we have our own preferences. I might value a lens’ colour rendition more than its sharpness, you might be the opposite. Someone else might value ergonomics over IQ, etc. That’s fine, and we’ll live with that because we have to. Being honest about one’s subjectivity is important (while feigned objectivity is just plain misleading). For instance, you might notice that Pekka has a lot of Nikkor lenses, and it is true that Pekka’s ‘collection’ is weighed towards Nikkors, but that does not mean that Pekka would not be able to say when a Nikkor lens disappoints him.
Conclusion
Well, that’s basically it, except for one last thing: another aspect of our humanity is our capability to err. In this, we will quote AnandTech, which is one review publication (albeit, in another field), that we have come to respect for its quality, integrity and continuing voyage of improvement:
“We are human. We make mistakes. We gladly welcome criticism from our readers and vendors alike. Seeking perfection doesn’t mean being perfect from the start, it means being able and willing to improve when faced with evidence that you’re not perfect. We feel strongly about this – negative feedback is tough to hear, but as far as we’re concerned it’s free education.“
(Quoted from About AnandTech, subheading review philosophies, on the 19th of October 2020)
Feel free to educate us.
The JAPB team