Conducting a systematic review

For the first part of my PhD my supervisor and I agreed that it would be good to do a systematic review, after identifying that the research area I am studying was in dire need of one. I have mentioned the systematic review I conducted in a previous post, and that it took me quite some time, during which I felt very unmotivated and somewhat ran out of steam. However – once I had recharged my batteries and rekindled some enthusiasm, I got right back on that Systematic Review train and got it finished. And I can honestly say, I am really glad I did it. It has provided me with not only the knowledge and experience of conducting a review, but has also helped massively in terms of situating my own research within a body of literature that already exists.

If you are considering conducting a systematic review, or are about to embark on one, read on to find out more about my process and some of my top tips. These may also be helpful for carrying out a general literature review too.

Be clear on your research question and rationale

It’s true what they say (‘they’ being lecturers that is), that a research question is vital, and it is certainly so when you start out on the journey of a systematic review. What you need to bear in mind is that your research question will be the core of everything you do. You will need to keep coming back to it and asking, does this answer my research question? Is this helping me address my research question? Being clear on this means that when you are faced with a tricky decision about whether to include or exclude a paper, you can return to your question and hopefully the decision will be made.

Be super clear on your inclusion and exclusion criteria

Talking about making those decisions when it comes to including a paper in your review or ditching it, you need to be extremely sure of your exclusion and inclusion criteria. Once these are in place, there is no budging. These are you parameters. Each of these criteria need to be in place because they either help you address your research question and overall aim, or they have nothing to do with it. It can be so hard to exclude some papers, just because they are interesting or add a novel idea to the whole literature area, but that doesn’t mean it can be included in the review itself if it doesn’t address your research question. It’s worth holding on to if it’s useful for your introduction or discussion though. Later down the line you’ll need to bring it all together and you need to have solid reasons for including a paper.

To help myself with this I whittled down my criteria and then I made a table with a justification for each one. I was then able to come back to this and remind myself why these particular criteria were in place and it helped me make decisions. Sit down and ask yourself why that criteria matters, justify it so that if you are challenged on it, or by a particular paper, you can argue your case.

List of inclusion/exclusion criteria with justifications

Keep a really accurate record of everything

Every single search you do, every single word or search term, every number of papers you find. When/why/how you include or exclude papers. If you are planning on publishing your findings you will need this information to prove that you conducted your systematic review rigorously. I had a document of the ‘systematic review process’ where I recorded each and every step I took. This meant I knew exactly how many papers were in my review at any one time, at which points they were excluded and why, and where those papers had come from.

Detailed record of the systematic review process

It’s really handy to do this because at the end of the process you will want to produce a ‘flow diagram’ of some sort to show how your papers were selected and if you have this information you just pick it up and input it into your diagram!

Diagram of the paper selection

Follow guidelines

There are a number of different guidelines out there for conducting a systematic review. Again, this is important if you want to be published, as it ensures that your review is of a particular standard. The PRISMA guidelines are very popular, and most journals ask for this to be completed when submitting a paper that is a systematic review or meta-analysis. I found this really useful as it kept me on track and it provides a really straightforward way of ticking off different tasks and making sure you are doing things correctly.

PRISMA checklist – one of many guidelines available online

Have a plan

Guidelines like PRISMA are kind of a plan in themselves, but it is really useful to look up other strategies and make your own plan. I sought out a number of different articles online about conducting a review, made my plan and then stuck to it. Again, this meant I had evidence of my process later down the line.

My plan looked something like this, and it was what my ‘systematic review process’ document was framed around.

  1. Research question and aim formulation
    Deciding exactly what the research question would be and what the ‘point’ of the systematic review was.
  2. Search term formulation
    Deciding which search terms were relevant, whether I needed to consider terms that were used elsewhere in the world. For example in the UK ‘carer’ is popular, in the US ‘caregiver’ is more so.
  3. Inclusion and exclusion criteria development
    Deciding what the papers needed to include and on what basis they would be excluded. For example I was investigating informal/family caregivers, so those investigating formal caregivers such as nurses were to be excluded.
  4. Searching and locating studies via online databases
    Deciding which databases to search and then conducting the search using the appropriate search terms and Booleans (AND/OR/NOT)
  5. Scanning titles for exclusion/inclusion criteria
    What it says on the tin!
  6. Scanning abstracts for exclusion/inclusion criteria
    And again!
  7. Scanning full texts for exclusion/inclusion criteria
    And again!(It’s steps 5, 6 and 7 that can become rather tedious and soul destroying, particularly if your initial search found a lot of papers. Hang in there!)
  8. Scanning reference lists for potential relevant papers
    Of the papers included in the review scanning the reference lists to see if they had cited any additional papers the search did not pick up, and then applying the inclusion/exclusion criteria to see if they were suitable for the review.
  9. Contacting experts for potential relevant papers
    Sometimes people can recommend papers that again haven’t shown up in the review, or they may have papers due to go to press which you might want to keep an eye out for.
  10. Quality criteria checking for each included paper
    Each paper needs to be checked for its methodological quality. Finding quality checking criteria can be difficult but there are various different versions available depending on the types of studies you are reviewing (e.g. longitudinal, RCT, cross-sectional). Once the quality has been determined, you can decide if you want to include them in your review or not.
  11. Concordance checking
    You need to get someone on board who is willing to cross check some of the papers you have both included and excluded. Sending a selection of both and asking them to look at the inclusion and exclusion criteria helps you check that you haven’t been on the wrong lines or accidentally excluding something that you should have included or vice versa.
  12. Data extraction
    This involves going through all the papers included in your review and extracting the important data. For me this included the authors, paper title, participant characteristics, study design, measures employed and the findings. Pop it all into a ‘data extraction’ table for easy reference.
  13. Data synthesis
    Now is the time where you have to try and bring together all the findings of the papers you have selected. Hopefully there will be some common themes and you can provide evidence for them, and depending on whether you are have included quantitative studies, qualitative studies or both, the way in which you do the synthesis may differ. For me, I like to have things in front of me and am quite visual. So it was helpful to print off my data extraction table and go mad with highlighters and annotations to colour code where I could see patterns.
  14. Write up!
    Once you have your findings it’s time to write up. Again, using the PRISMA guidelines for this is useful because you pretty much have a guide of everything you need to include for a rigorous paper.

I am by no means an expert on systematic reviews but these are things that helped me when finding my way with my first ever one. It was a really long, arduous process and at times seemed never ending. But as I said, I learnt a lot and my field desperately needed a review in this area.

Let me know if you have conducted a systematic review, how you found it and whether you would add anything to this post. If you are facing a systematic review at the moment or in the future, all the best with it!

Til next time 🙂


One thought on “Conducting a systematic review

Leave a Reply