Large language models (LLMs) and Retrieval-Augmented-Generation (RAG) show remarkable capabilities in Open-domain question-answering (ODQA). Despite the advancements, LLMs tend to generate verbose responses, of which only a small part is the answer phrase. Although the ability to produce the confidence score for the answer is essential when deploying LLMs in high-risk domains, sequence probabilities obtained from LLMs do not correlate well with the probabilities of correctness and thus fail to represent confidence scores. This study introduces Answer-prefix Generation (Anspre) to improve generation quality, allowing the LLMs to output answer phrases and produce highly reliable confidence scores. We guide the model in predicting the answer phrase using an answer prefix and design a ranking score that integrates parametric and non-parametric knowledge. The answer phrases and their corresponding scores enable Anspre to aggregate results from different documents and samplings to boost performance and produce confidence scores highly correlated with correctness. We show that Anspre can be applied to any LLM and present an approach called Self-Anspre to combine Anspre with Self-reflective RAG, a state-of-the-art framework based on reflection tokens. Empirical evaluation on popular ODQA benchmarks shows that Anspre and Self-Anspre significantly improve state-of-the-art LLMs and RAG frameworks. An in-depth analysis shows that confidence scores produced by Anspre are highly correlated to the likelihood of correctness.