Evaluations of an entrepreneurship development program: A systematic literature review

The contribution of entrepreneurship at both the microeconomic and macroeconomic levels is remarkable. There are many entrepreneurship development programs (EDP) initiatives proposed by policymakers around the world to ensure the growth of entrepreneurs. While EDP is necessary to improve entrepreneur capabilities, however studies on EDP evaluation literacy remain limited. This study aims to fill the gap by investigating EDP evaluation literacy by using a systematic literature review (SLR). This study carried out a bibliometric analysis on a Scopus database in the last three decades, from 1989 to 2021 with all the manuscripts were written in English. The results show that most of the literature on EDP evaluation-related themes is mainly focused on program evaluation, characteristics, challenges, stakeholders, and context, with program evaluation and characteristics are considered as the most significant issues of EDP evaluation. Interestingly, this study also reveals that few manuscripts review the process of EDP as the success criterion of EDP implementation.


Introduction
Entrepreneurship development programs (EDP) are activities (or a package of training and counseling designed) to increase knowledge and creativity. EDP encourages the entrepreneurial spirit of people to start and grow their business in both formal (high-school and post-secondary school students) and informal sectors (Gangi, 2017;Hägg & Gabrielsson, 2020;Lackéus, 2015;Valerio et al., 2014). The purposes of EDP are to improve skills, knowledge, and attitudes of targeted individuals and community (Fayolle et al., 2016;Gedeon, 2017;Klingler-vidra et al., 2019), to start a new business, and to add new jobs (Decker et al., 2014;Jeng & Hung, 2019), to provide an alternatives sources of income (Dvouletý, 2017;Terjesen et al., 2016), and to initiate economic growth (Martínez et al., 2018;Song & Winkler, 2014;Stoica & Roman, 2020). Nowadays, EDP implementation is getting more recognition and attention from policymakers and academicians because of its contributions (Tsai et al., 2014). EDP offered many benefits to the entrepreneurial community. It increases the economic contribution, gives alternative solutions to youth unemployment issues, creates social impact, etc. (Nabi et al., 2017;Pettersson et al., 2017;Valerio et al., 2014).
However, Gustafsson-Pesonen & Remes (2012) argued that despite its popularity, it's very often reported that the biggest obstacles of entrepreneurship programs are on what (and how) educators or trainers teach and their attitudes toward entrepreneurship. Besides, Gustafsson-Pesonen & Remes (2012) also pointed that the evaluation of the EDP effectiveness remains arguable. Fayolle (2013) added that while the impact of the program or session on trainees, students, and other participants may be an appropriate evaluation criterion, there is still the issue of how to measure it. What should the indicators be, and how should they be measured?
How can you quantify a shift in someone's intention or behavior? How may the sign of the time component be considered? And how can educational, teaching, and training aspects be distinguished from all other factors that influence the decision to pursue a particular career path or profession?
The clear EDP evaluation measurement is necessary to provide tremendous program improvement of all deficiencies in achieving the program's goals and objectives and closing the loop of the program decision-making. Although there is still debate about evaluation measurement due to the absence of a control group (Fayolle, 2013) and meta-analysis (Bae et al., 2014), this problem can be minimized by analyzing its nature, educational interventions, and context. Therefore, to fill such a gap, the study aims to explore the state of EDP evaluations using a systematic literature review (SLR) since this method can give an exceptional knowledge of the current body of work (Frank & Hatak, 2014). We have investigated 366 papers from the Scopus database that were published from 1989 to 2021 that used the keywords "entrepreneur," "develop," "evaluate," "education," and "training." The usage of Scopus as the primary database in the study is because this reputable international database is utilized by academicians worldwide. By investigating state of the art, we identified five main themes: evaluation, program characteristics, challenge, stakeholder, and context. Besides these central themes, we also evaluated the manuscript based on the period it was published, the contributors, and the subject area of the manuscript. This study is considered the first amongst EDP-related manuscripts that uses SLR to explore the evaluations of the EDP. Furthermore, the systemic literature review is used as this method can create an academic map regarding EDP evaluation trends and futures (Alvesson & Sandberg, 2011).
The manuscript is composed as follows: the first and second sections will briefly review academic literature, concentrating specifically on the theme's critical aspect of EDP.
Subsequently, the third section states the methodology of the research to produce a duplicable SLR. Following these come the results of the SLR and a review of EDP trends constructed in journals studied of this theme. Thus, the final part contains the result, suggestions, limitations, and suggestions for future research.

Entrepreneurship Development Program (EDP)
Entrepreneurship is considered a critical alternative strategy to deal with various situations, such as financial crises and increased unemployment rates (Galvão et al., 2018). It is appreciated as a trigger of regional prosperity and social development (Ashrafi et al., 2020;Galvão et al., 2019;Song & Winkler, 2014). The entrepreneurship level of a country has been shown to influence the advancement of national economies (Atems & Shand, 2018;Prieger et al., 2016), progress and innovation (Landström & Harirchi, 2018;Sánchez, 2013), the number of job creation (Shane & Venkataram, 2012), and the growth in nominal Gross Domestic Product (Khyareh et al., 2018). Therefore, due to the critical contribution to the  , 1998;Gabrielsson et al., 2020;Heinonen & Hytti, 2016;Interman, 1992;Kyrö, 2015;Lackéus, 2015) classified these education and training programs into simple three purposes. The purposes are to promote business awareness and creation, small business development, and train trainers.

Entrepreneurship Development Programs (EDP) Evaluation
The attention of EDP is booming worldwide. Although there is debate whether these programs should be introduced at an early age (elementary school) or university level, more and more public policy "initiatives" related to entrepreneurship are implemented by the government to accelerate the growth of business and enhance entrepreneurial spirit and literacies. (Fayolle, 2013) also added that given that it is a hot topic on the political agenda, there is an increasing number of publications by academicians that have focused on EDP.
However, the critical question now arises, whether the implementation of EDP affects the intended goals and produces an effective result. (Brentnall et al., 2018) argued that if public money were allocated on such initiatives because they are represented as solutions to societal issues, it was reasonable to examine whether they were a good investment. Therefore, more investigations should be made in evaluating EDP (Farashah, 2013;Larso et al., 2018).
Surprisingly, despite the huge number of public initiatives allocated to this program (and the critical issue from government perspectives), a little study is available to comprehensively discuss the assessment and evaluation of EDP (Fayolle, 2013). Yet Acs et al. (2016) and The nature of EDP that is inseparable from training and education made the teaching model one of the critical sources in evaluating EDP. Eseryel (2002) identified six approaches that can be used in assessing EDP: goal-based evaluation, goal-free evaluation, responsive evaluation, systems evaluation, and professional review and quasi-legal. Fayolle (2013) added that among these approaches, goal-based (Kirkpatrick's reaction, learning, behavior, result framework) and systems-based approaches (such as Input, Process, Output, Outcome model by Bushnell) are predominantly used to assess EDP. EDP evaluation should focus more on the process than content (Gruenwald, 2014). Lundström et al. (2014) and Vesper & Gartner (1997) also strengthened that argument by stating that the accomplishment of programs goals and teaching methods are the two most commonly used to assess EDP. used internally for quality control and improvement purposes (Lundström et al., 2014;

Methods
This study used systematic literature reviews that cover EDP evaluation-related topics for program improvement. Armitage & Keeble-Ramsay (2008) Borrego et al. (2014) and Tranfield et al. (2003) added that the SLR technique shows an important practical (in terms of accuracy for duplicability) when dealing with many investigations presented in a long period.
SLRs are widely used for conducting evidence-based policy (Pittaway, 2016;Pittaway & Cope, 2007). Therefore, considering these points, we utilize SLR to identify issues on EDP evaluation and analyze its growth. The analysis also intends to assess the impact of papers on this subject, analyze journals that produce documents, verify significant writers and academicians, and classify desirable topics and procedures.
The search was conducted using the Scopus database and covered the period 1989 to 2021. The terms used to determine relevant studies were examined in titles, abstracts, and keywords. These terms were entrepreneur*," "develop*," "evaluate*," and "education or training." The core logic to select the Scopus database was its extensive analysis of universally indexed papers of a value standard by academics. Furthermore, many papers recognized by Scopus are also accessible in other databases. As one of the most comprehensive databases, Scopus can also form a different bibliographic outcome than other databases. Because the research of EDP evaluation remains thin, we chose to use all articles about this topic in Scopus, from 1989-2021, to have a rich data source. To the best of our knowledge, there is no existing SLR that focuses on exploring the evaluations of the EDP from the Scopus database published from 1989 to 2021. From this search, it can be known that the first papers in the Scopus were written by Wan (1989) and Curran & Stanworth. (1989) While Wan (1989) study focused on evaluating the type and efficacy of a company's workshop program to promote business planning competencies, Curran & Stanworth (1989) assessed the character of entrepreneurship education. For our study, these two papers published in 1989 will be treated as the beginning point of the EDP evaluation discourse.
Furthermore, we first selected documents with analytically confirmed information and eliminated documents other than journal articles to obtain the necessary papers. Then, we chose "entrepreneur*," "develop*," "evaluate*," and "education or training" as keywords to discover papers related to this theme. The resulting outcome was a total of 1191 papers.
These papers then undergo four phases of elimination. First, we eliminated papers not written in English (reducing the article count from 1191 to 1132). Second, writings other than social, management, and business scope were eliminated, reducing the papers to 481. Third, we excluded other disciplines to focus just on the theme. Lastly, we stopped outcomes that were not a journal article or different than the theme. The final result used as the base for SLR is 366 papers. Table 1 presents a summary of the phase used in the SLR method.

1191
Stage 2 Choosing outputs that were written in English. 1132 Stage 3 Choosing output that focused on social, management, and business scope. 481 Stage 4 Choosing the scientific papers only, eliminate conference papers and review. 366 The quality impact of the manuscript often refers to the number of citations. Although the usage of citation has been a debate for over decades (as the older the manuscript has had more time to accumulate citations), the usage of citation as the proxy of the quality of manuscript remained still (Dumay et al., 2014). Therefore, to overcome the bias, this study uses citation analysis into three points of view which are co-authors citations network analysis, co-citations sources network analysis, and co-citations and corresponding group analysis. The result of Tables 3 and 4 compare the top 10 authors and top 10 co-cited authors indicated that rather than a few authors dominating the rankings, the rankings show a wide range of authors; this reflects the breadth and diversity of scholars and institutions working in this field (Demartini & Beretta, 2020;Massaro et al., 2016). Furthermore, the analytical framework and dataset (text queries) must be defined and coded using the developed framework as another phase of SLR (Massaro et al., 2016). Following the coding process, we completed the final phase of the SLR, which involves analyzing the results to produce insights and critiques, respond to the research question, and make recommendations for future research directions. Figure 1 shows the summary of the SLR process used in the study. other than participants (employers and academicians). He found that the training program has effectively achieved the program's goals to promote business planning competency. The evaluation of these enterprise's workshop results is that these programs lack focus on personal training qualities, as they over-emphasize practical skills in business planning.

Entrepreneurship Development Programs (EDP) Research Trends
Following Wan (1989) and Curran & Stanworth (1989), the study concerned four typologies of entrepreneurial education: entrepreneurial education for small businesses, entrepreneurial awareness for entrepreneurs, advancing industry for small business owners, and small business awareness. They evaluate that the program with the highest effectiveness was small business education for small business owners. cumulative with the contribution of more than 50%. Europe accounted for the biggest continent contributors of the study related to EDP with a contribution of 45% followed by   While co-author in this study is defined as one or more persons who write the article, cocitations are the frequency in which two documents are cited together by other documents.  Heinonen, Jarna 28 183 6.5

The Citation Network Analysis
The Citation Network Analysis (CNA) is based on the list of references found in journal articles or other types of publications. References are citations to prior works that have influenced the direction of a study. Even though papers with a large number of citations are not necessarily representative of influential and world-class research, it is widely accepted that citations can be used to assess the relevance of publications (Colicchia et al., 2018;Dawson et al., 2014). Papers may be removed from the study if no other works cite them, regardless of their content's importance, or if they have garnered a small number of citations due to their recent publication. Other technologies, like keyword analysis and citation score analysis, might thus complement and minimize the aforementioned flaws. Therefore, to build bibliographic networks, a VOS viewer was used to analyze bibliometric networks and cluster analysis. Figure 3 presents the co-author's network analysis based on (at least) 20 co-citations per author, demonstrating the "world" of five groups of the most outstanding co-authors of 106 authors. Although these authors discuss a diverse range of topics, they cite one another quite frequently. It signifies the presence of a solid network (correlation) between them. It also can be acknowledged that the EDP study is growing by citing each other's papers.
Furthermore, Figure 4 that was formed with at least 20 co-citations per journal shows that   are also discussed in this group as Fiet (2000aFiet ( , 2000b argues that the teaching process should not be dull, as the most effective programs for coaching theory are connected to definite persons, places, timing, contacts, and environments.

Group 2
This group consists of 14 list manuscripts with seven of them having more than four co- Furthermore, the topic of program characteristics was discussed by Fayolle & Gailly (2008). There are three categories of entrepreneurship programs namely learning to be enterprising persons, learning to be competent entrepreneurs, studying to be academicians (teachers or researchers in the field of entrepreneurship). Another study from Gibb (2002) and Gorman et al. (1997) even added that as the rising of entrepreneurship interest is mainly due to the pressure of globalization, the need for entrepreneurial behavior is being felt by a broad variety of stakeholders (e.g., pastors, medics, educators, communities, and others) with a concern on the market.
The first paper studied by Solomon (2007)  proposed that the prediction model should depend on the intention because situational or individual determinants have a low prediction ability based on the empiric cases. Finally, Thomas & Mueller (2000) investigated culture or the connection among philosophy on four personality traits that are linked with entrepreneurial passion. The personal entrepreneurial characteristics are innovation, locus of control, risk-taking, and energy.

Group 5
This group contains 14 manuscripts that emphasize EDP evaluation, characteristics, and challenges. Similar to the previous group, most of the manuscripts in this group concurs that EDP evaluation can be and should be conducted based on the impacts (Athayde, 2009;Fayolle & Gailiy, 2015;Graevenitz & Weber, 2009;Liñán et al., 2011;Nabi et al., 2017). proposed the elements forming an entrepreneurial intention. They stated that personal characters and perceived behavioral control are the determinants of entrepreneurial intention.
The second topic is program characteristics. Kuratko (2005) and Pittaway & Cope (2007) studied the contents of EDP which is the contents of market entry, compromise skills, leadership, construction skills, creative problem solving, and skills to use technology and innovation skills, intention to be entrepreneurs, business capital, copyright, risk management, characters of an entrepreneur, and growth constrain. The entrepreneurship education contents are established based on the real-life situation: business strategy, creating business for scholars, discussion with industrialists, situation investigation, and case study.
Thirdly, the program challenges topic. Kuratko (2005) divided the challenges of entrepreneurship study into ten issues namely stagnation, low quality of publications, lack of academician experts in entrepreneurship, low level of technology usage, the dot-com tradition, conflicting of academia with the real-life, the dilution of entrepreneurship, managing the risk, the leadership revolving of academic administrative, and the one-man show. Therefore, to overcome the challenges in promoting entrepreneurship programs, the Evaluations of an entrepreneurship development program: A systematic literature review by Ronal Ferdilan, Wawan Dhewanto, Sonny Rustiadi usage of robust theoretical and conceptual basics combine with a real investigation (not take it for granted) is a mandatory act (Fayolle, 2013).

Discussion
The descriptive of EDP issue from the five selected groups above is mostly concerned with program evaluation, characteristics, challenges, stakeholders, and context. In addition, the topic of program evaluation and program characteristics are recognized as the main significant them of EDP evaluation (See Table 5). In addition, the findings show that most evaluations are conducted based on the impact of the program. Just a few of the papers consider the process of EDP as the assessment of the EDP implementation.  Table 5 above illustrates the five topics of co-authors in the EDP-related studies. From the existing papers, several topics are discussed, with some of them revealing similarities.
The most significant issues are program evaluations and program characteristics, found in four groups. Firstly, EDP can be evaluated based on impacts and processes (although most of the assessments also assess the implications to assess the program achievement). To appraise the result of the program is necessary, but to assume by its impact alone is significantly insufficient, as the goal of the review is to improve the program. The findings above also reveal that a few articles review the process. Nevertheless, since two other minority authors use the process as a basis of a program evaluation, it can be considered as rebalancing the program evaluation and identifying current and future trends.
There were five topics discussed in the literature. The first topic was the EDP evaluation.
The literature on EDP evaluation reveals that EDP evaluations are primarily concerned with the impact of the program on the participants' capabilities and their business. Nevertheless, just a few of the bibliography suggested for review on the EDP implementation process, such as the delivery process, the expertise of stakeholders involved, and other aspects of the implementation process. Even though most previous research conducted the EDP evaluation based on the impact, this study expands the scope for EDP evaluation based on both effect and the process. The aim to evaluate the impact is to find whether the intended goals are achieved or not. Besides, the objectives of the review of the operation of the EDP are to find out the quality of the process and improvement purposes. The findings conclude the assessment typology based on both the impact to see the result and the process that consist of the evaluation of the impact and process, the evaluation of the impact, and the evaluation of the process. However, since the EDP evaluation is complex due to the variety of the content, purpose, and approach (Fayolle & Gailiy, 2015), the EDP evaluation should be conducted to ensure the program enhancement (Warhuus et al., 2017). This EDP evaluation and who can influence or be influenced by business activities. There are more than ten stakeholder categories that were defined by Leonidou et al. (2020) which consists of academia, government, community, customers, service intermediaries, user community, suppliers, business networks, start-up teams, innovation intermediaries, social media, university peers, NGOs, industry cluster, associations, friend and family. However, these wider ranges of stakeholders have often produced contra-productive conflict between each other due to the different systems and interests of each stakeholder. Some authors use Quadruple Helix (Dhewanto et al., 2021;Primanto et al., 2018), the Penta Helix (Tonkovic et al., 2015), collaboration, and the individual actors' model (Rustiadi & Arina, 2019) as a framework to analyze the roles that stakeholders. Finally, the last issue of EDP evaluation in this study is the context, which means the diversity of programs.
Evaluations of an entrepreneurship development program: A systematic literature review by Ronal Ferdilan, Wawan Dhewanto, Sonny Rustiadi

Conclusion and Suggestion
In this literature review, we have investigated 366 EDP manuscript related and explored several issues such as program evaluation, characteristics, challenges, stakeholders, and context, with program evaluation and characteristics. This study shows that most manuscripts that conducted EDP evaluations are based on the program's impact on the participants' competency, their businesses, or their career. While a few that the review of the process of EDP as the success criterion of EDP implementation. This study helps to identify EDP evaluation criteria based on their impact and process so it can produce a more comprehensive evaluation to improve EDP implementation. From a practical perspective, this study shows the policymaker the benefits of EDP evaluation. Other EDP studies may also be inspired to conduct EDP evaluation to make the program more effective and efficient. In summary, to accomplish the goal effectively and efficiently, there should be an evaluation of the goal's achievement and the antecedents.
The limitation of this study is the use of the Scopus database by excluding some papers in which involves subjectivity in grouping the articles and choosing the keywords. Future research could include Web of Science bibliographies, books, book chapters, and conference papers as sources. Future studies could also be an investigation on the evolution of the EDP implementation and evaluation during . It would also be compelling to discuss the conduct of the evaluation of EDP within Asia, Australia, and Africa contexts. Furthermore, future studies should also consider the need to explore, reflect, and ask questions more critically, not on the side of being taken for granted.