The IIAR “Tragic Quadrant”

By / / 451 Group, Constellation, Digital Clarity Group, ESG, Forrester, Gartner, Gartner Magic Quadrant, HFS Research, Humour, IDC, IIAR>, IIAR> Analyst Of The Year, IIAR> Tragic Quadrant, IT Analysis Firms, SMB Group, The IIAR> Analyst Of The Year (AOTY), Ventana

Last year, as part of the 2014 IIAR Analyst of The Year Survey, we invited analyst relations professionals to rate their favourite industry analyst individuals and the firms they worked for. More than 60 individual organisations responded to our survey. We were interested to see if we could do further analysis on the data that was collected.

When we set out to do the IIAR Analyst of the Year (with Helen Chantry), we always had envisioned doing a Magic Quadrant of analyst firms. This year the survey provided us with further information which we have been able to breakdown and analyse to provide a more detailed understanding of how analyst relations professionals perceive the relevance, impact and reachability of industry analyst firms. We are not claiming that this is an exhaustive study. Rather it simply opens a new (slightly cheeky – hence the notion of “Tragic Quadrant”) window onto the analyst landscape, where we attempt to rank industry analyst firms by impact, relevance and ease to do business with.

The IIAR survey asked respondents to assess individual industry analysts on a range of issues broadly grouped according to the following criteria:

  • ‘Impact’ (to what extent does the industry analyst impact the technology purchase decision);
  • ‘Relevance’ (is the particular industry analyst relevant to the technology purchase decision, do they understand the marketplace);
  • ‘Interaction’ (how easy is the industry analyst to reach and to interact with).

We are able to represent the top 13 industry analyst firms according to this new analysis. We have produced three Tragic Quadrants according to i) overall industry analyst firms, ii) global industry analyst firms, and finally iii) independent industry analyst firms.

  • The Y axis depicts the ‘Impact’ of the industry analyst firm on the purchase decision. This also relates to their perceived credibility and capability to provide an objective opinion.
  • The X axis marks their ‘Relevance’ for the purchase decision. This means their capability to cover the market and their specific geographical allocation. It also includes public recognition of their presence in the market (e.g. as an expert).
  • The size of the bubble is ‘Interaction’. This relates to issues of communication (e.g. how easy is it to get to them and to talk to them).

Overall Industry Analyst Firms Quadrant

The IIAR Tragic Quadrant 2015 (Overall) featuring Gartner, IDC, Forrester, Ovum, HfS, Constellation, 451 Research, Celent, Pac, ESG, Digital Clarity Group, Ventana, SMB Group
The IIAR Tragic Quadrant 2015 (overall) featuring Gartner, IDC, Forrester, Ovum, HfS, Constellation, 451 Research, Celent, Pac, ESG, Digital Clarity Group, Ventana, SMB Group

In terms of the Overall Industry Analyst Firm Quadrant, Gartner, Forrester and IDC came out ahead of rivals. In terms of their specific ranking according to our analysis, Gartner is 1st, Forrester 2nd, and IDC 3rd. Gartner is higher on impact and relevance than its immediate rivals, but did less well in terms of interaction. IDC by contrast scores someway ahead of Gartner in terms of interaction but is perceived to be slightly less impactful and relevant. By contrast, although some way behind the leaders in terms of impact and relevance, Digital Clarity Group, Ventana and SMB Group did particularly well on interaction.

Global Industry Analyst Firms Quadrant

2015 IIAR Tragic Quadrant - Global Firms featuring Gartner, Forrester, IDC, HfS Research, Ovum, 451 Research
2015 IIAR Tragic Quadrant – Global Firms featuring Gartner, Forrester, IDC, HfS Research, Ovum, 451 Research

In terms of the Global Industry Analyst Firms Quadrant, Gartner stood out as 1st. This was followed by, in 2nd place, Forrester, and IDC in 3rd place. As mentioned above, Gartner were successful with high levels of relevance but performed less well on interaction. IDC performed particularly well on impact and interaction. In terms of the other firms, HfS performed well across all three categories but scored particularly well on impact. 451 Research and Ovum did well based on interaction.

Independent Industry Analyst Firms Quadrant

2015 IIAR Tragic Quadrant - Independent analyst firms featuring Constellation, HfS Research, Ventana, SMB Group
2015 IIAR Tragic Quadrant – Independent analyst firms featuring Constellation, HfS Research, Ventana, SMB Group

In terms of the Independent Industry Analyst Firms Quadrant, Constellation were in 1st place, HfS in 2nd place, followed by SMB Group in 3rd place and Ventana in 4th. Constellation scored particularly well on impact, whilst HfS performed well on impact and interaction but less well on relevance. SMB Group scored highly in terms of interaction. Ventana did well in terms of impact but slightly lower in terms of interaction.

In case you were wondering the Tragic Quadrant is by no mean meant to be a normative or scientific depiction of the industry analysis industry. It offers but one window onto this world that was partially inspired by Juvenal’s ancient dictum Quis custodiet ipsos custodies? (which, roughly translated, is who assesses the assessors? or, in our case, who analyses the analysts?). Whilst it was initially intended as a bit of fun we do nevertheless think there are some interesting aspects for AR professionals and analyst houses to take from it. We see at least two takeaways emerging for different audiences:

  • AR professionals may use this as a subjective analysis tool to look at their target audience engagement strategies. They should balance ‘ease to do business with’ against ‘relevance’ and ‘impact’. Or, in other words, they shouldn’t brief analysts just because they’re easy to deal with (or conversely they should look at analysts which are less of a pain depending on the type of impact the AR professional is looking for (see the AR SOSM model);
  • analyst firms should monitor the ‘transactional tax’ they impose on AR people: if they raise the ‘interaction barrier’ too high while not providing sufficient coverage and not showing impact, their vendor information source might soon provide them only a partial view of the market (raising exhaustivity and fairness issues) or their vendor revenues might suffer too.

 

By Neil Pollock (LinkedIn, @neilpollock), Yulia Sidorova (LinkedIn) and Ludovic Leforestier (LinkedIn, @lludovic)

 

Related posts:

36 thoughts on “The IIAR “Tragic Quadrant””

  1. Stephen England

    We and our clients have almost zero interest in which firms vendors find easy to deal with. What we want to understand is which firms the end users find easy to deal with and look to for advice.

    KCG’s Mystical Box Charts plot precisely that, and our annual revenue analysis and landscape backs that up with the numbers that matter – what end users spend on advice from the firms – advice on what to buy, who to buy it from and what to pay for it…

    Our analysis does not align with this at all except for Gartner being number 1. Gartner takes well over 70 cents of EVERY SINGLE end user dollar spend on RAS.

  2. Michael Rennett

    Does the IIAR have a breakdown of the type of AR professionals that answered these questions? It would be interesting to see if there was a particular skew based on coverage area – ie. if the vast majority of respondents worked in IT Services AR, or if there were no AR professionals that covered the mobile industry.

    Realise this is all subjective but it could give some good insight on the above.

  4. Duncan Chapple

    The strength and the weakness of this quadrant is that it shows how vendors over-estimate a few analyst firms. I’ve just posted an article to explain why here: http://bit.ly/1Gdb3dQ

    Stephen is right that it’s influence on purchasing that counts, but it’s been ten years since you could equate the buyers of research with the buyers of IT and telecoms solutions. There’s hard data that shows that, and I’ve written about that too here: http://bit.ly/1dwzb12 Given the impact of freemium research and vendor-sponsored analysts, I just cannot say that 1000 Gartner analysts have 100 times more influence than every other time. That means that a representative group of 10 Gartner analysts have more influence on end-users than the 6,500 non-Gartner analysts. That’s just silly.

    The Mystical Box, like Stephen’s market sizing of the analyst industry, have a lot of necessary guesswork behind them. Whatever the weaknesses, is has the redeeming effort on trying to focus on something which is helpful to AR people: focussing them on where they can made a business impact. Surely the logic of highlighting firms that are easy to work with would be that you’d spend more time on them, and less time on the analysts who are more influential but harder to work with.

  5. Stephen England

    “Surely the logic of highlighting firms that are easy to work with would be that you’d spend more time on them, and less time on the analysts who are more influential but harder to work with.” said Duncan

    That’s the entire point – focus on influence not ease of interactions.

    And, if you judge influence by where end users actually spend money all that has changed in the last ten years really is the level of noise.

  6. Pingback: Cottrill Research » The IIAR Tragic Quadrant

  7. Duncan Chapple

    Ludovic, honestly AR is not a major sale channel for most analyst firms. As you and I both say, if analyst firms want to develop business relationships with AR people then they should pay attention to how easy or hard it is to work with them. But not every analyst firm is going to go soft on vendors for that reason, especially if the analysts are disconnected from business development.

    Stephen, I totally agree: focus on the analysts that matter, and accept that some interactions will be hard. But, indeed, the reason why you see freemium and vendor-sponsored research as noise is because you mistakenly equate analyst influence with end-users spending on analysts. Even in organisations that do subscribe to Gartner and Forrester, there are often many more people using freemium and vendor-sponsored research. The days are over when AR people can ignore the long tail.

    1. Ludovic Leforestier

      Duncan, do you have facts to back that up? I’ve worked in about 10 companies over the last 15 years and I can assure you that AR is a key influencer and/or buyer for research.

      Stephen, Duncan: the long tail is important for some AR folks, maybe not for others. It’s a question of alignment, I find strange that so few AR folks get this. See also the AR Compass > http://iiar.wpcomstaging.com/2013/05/10/iiar-best-practice-webinar-the-ar-compass/

    2. Stephen England

      Our CIO clients strongly disagree. The small firms are interesting but there is no proof they influence the Fortune “10,000” at all in our opinion. SMB is changing for sure though.

      They would love to have us believe otherwise.

  8. Laurie McCabe

    Couple of thoughts here. First, as a very small firm, we’re honored just to be on the radar! Second, I believe we’re the only ones that focus exclusively on SMB. As we all know, most SMBs do not buy analyst services directly. Our main focus at SMB Group is to provide vendor clients with fact-based perspectives, consulting, etc. (based on our quantitative and qualitative research studies) to help them better understand, sell to and serve SMBs.

    That said we know “influence” is also key and are ramping up our efforts to increase distribution of our perspective to SMBs through more social channels and outlets. We welcome your thoughts on how we can improve and add more value!

  10. Pingback: HfS hits bullseye of the IIAR Tragic Quadrant for IT analyst firms - Enterprise Irregulars

  11. Chuck Van Court

    Any technology buyer who gives more than 5 to 10 percent weighting to any analyst firm’s evaluation is delinquent in their responsibilities to shareholders, team members and the folks who will be actually using the technology.

    Other than possibly being an effective CYA strategy for protecting purchasers from accountability for bad technology decisions in the eyes of naive board members, I have yet to see any analyst firm that does not suffer from two deadly sins to justify more than a 5-10 % weighting: They earn too much revenue from the firms they are writing about to be considered objective and unbiased and the level they actually use and understand the technology being reviewed lacks the necessary depth and practical usage.

    I ask you, should a person buying a car give much weight to a car reviewer who has at best only driven the car in a parking lot and who earns a large percentage of their income from the car manufacturers they are reviewing?

    Mind you, I have found most of the people in this business to be smart with generally good intentions, but in my mind they should stick to macro-level analysis and helping vendors understand how to best sell their offerings. Notice that I said they should help vendors understand how best to sell and not actually help them sell. Who does not really believe all the contests, matrices and watchlists are more about generating revenue from technology vendors and less about providing technology buyers with quality data points used in purchasing decisions?

    The best place to get quality data points used in evaluating technology is from hands-on usage and from in-depth reviews by actual users of the technology who are not financially or otherwise motivated to write favorable reviews, or at least keep their negative comments tempered toward the hand that feeds it. I have been told off-the-record by several analysts that technology purchasers often give too much weighting to their analysis, but the revenue is certainly too compelling to say that out loud.

    The problem is not with the people in this space, it’s with the business model. People inherently follow the money and who can really blame them?

      1. Chuck Van Court

        Hello Ludovic: I fully realize that, which only further reinforces the importance for analyst firms to fully disclose any benefits they receive from technology vendors. My god, the FTC requires more disclosures from an unknown blogger regarding potential conflicts of interest than they do from Gartner and the like. Do you think that is right? All should be disclosed and the buyers should be left to decide if it matters to them or not.

  12. Stephen England

    Absolutely terrifying that this group has analyst firm employees as members in my opinion – they are specifically banned from KCG Connects on LinkedIn so that our members can have safe open discussions. Please change the policy here.

    1. Michael Rennett

      The IIAR has member only discussions on the Huddle site, but as this is posted publicly on a blog then surely it’s within everyone’s rights to comment on it. The KCG Connects group serves a purpose but it can be beneficial to have analysts and AR professionals join the same debate. Disagree with your stance here Stephen.

          1. Stephen England

            I know blogs are public, that’s why we at KCG use a secure LinkedIn group. I do not disagree with your right to allow analyst firm employees but I strongly disagree with the fact that you do not WARN users that they should never discuss anything here that they don’t want the firms (lots of them) to read.

  13. Chuck Van Court

    Greetings: What is the hesitation in approving my comments (made Friday 12th June 2015 at 17:15) in this public blog? My comments are likely contrary to the opinions of many of your readers, but they are still respectful, detailed and touch on an important topic that warrants open discourse for the benefit of technology buyers. Thank you. Chuck

  14. Pingback: IIAR’s Tragic Quadrant Spotlights Overrated Analysts | Kea Company's Influencer Insights

  15. Pingback: Constellation and the curse of the quadrant | Institute of Industry Analyst Relations (IIAR)

  16. Pingback: The IIAR Tragic Quadrant for 2017 | Institute of Industry Analyst Relations (IIAR)

  17. Pingback: Who will be the IIAR> Analyst and Analyst Firm of the Year 2020? | Institute of Industry Analyst Relations (IIAR)

  18. Pingback: The IIAR> Institute of Influencer & Analyst Relations / Winners announced: IIAR Analyst of the Year 2016

  19. Pingback: The IIAR> Institute of Influencer & Analyst Relations / Winners announced: IIAR Analyst of the Year 2015

  20. Pingback: The IIAR> Institute of Influencer & Analyst Relations / Join us for the 2021 IIAR> London Christmas Networking party, celebrating the Analysts & Analyst Firms and the AR Professionals & AR Agencies of the year

  21. Pingback: The IIAR> Institute of Influencer & Analyst Relations / Announcing the IIAR> Awards 2021!

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: